Efficient Bounds for the Softmax Function and Applications to Approximate Inference in Hybrid models

The softmax link is used in many probabilistic model dealing with both discrete and continuous data. However, efficient Bayesian inference for this type of model is still an open problem due to the lack of efficient upper bound for the sum of exponentials. We propose three different bounds for this function and study their approximation properties. We give a direct application to the Bayesian treatment of multiclass logistic regression and discuss its generalization to deterministic approximate inference in hybrid probabilistic graphical models. The softmax function is the extension of the sigmoid function for more than two values. Its role is of central importance in many non-linear probabilistic models. In particular, many well-known models deal with discrete and continuous data. Variational approximations based on the minimization of the Kullback-Leibler divergence are one of the most popular tools in large-scale Bayesian inference. In recent years, generic tools such as VIBES [1] have been proposed for inference and learning of graphical models using mean field approximations. For graphs having discrete nodes with continuous parents, the direct mean field cannot be applied, since there is no conjugate family for the multinomial logistic model. Local variational approximations have been proposed in the case of binary variables [2]. They are based on a quadratic lower bound for the log of the sigmoid function. This is used in directed probabilistic graphical models having binary variables with continuous parents [3, 1]. However, for more than 2 categories, the inference problem remains unsolved. In section 2, give an detailed overview of the techniques used by different authors to deal with Bayesian inference for models involving softmax links. In section 3, we derive three simple lower bounds and discuss their approximation properties. In Section 4 we perform numerical experiments on a typical application which is the variational approximation of the multiclass logistic regression. Finally, we discuss its application to more general deterministic approximate inference algorithms in graphical models.